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Computer-Based  Assessment  of  Schema  Knowledge 
in  a  Flexible  Problem-Solving  Environment 


This  report  describes  assessment  of  students'  schema  knowledge  for 
arithmetic  word  problems.  The  assessment  takes  place  on-line,  as  students  interact 
with  the  computer  program.  The  instruction  that  precedes  the  assessment,  the 
environment  in  which  students  exhibit  their  knowledge,  and  the  assessment  of  that 
knowledge  all  derive  from  one  core  theory  of  learning  and  memory;  schema  theory. 
My  objective  here  is  to  describe  a  tightly  woven  system  of  instruction,  presentation, 
and  assessment.  All  three  parts  of  the  system  rely  on  computer  technology,  but  it  is 
particularly  useftil  in  assessment.  Because  all  of  the  components  of  the  system 
depend  critically  on  schema  theory,  I  shall  begin  with  a  brief  description  of  the 
theory  and  how  it  guides  them. 

SCHEMA  THEORY 

Schema  theory  describes  how  information  is  stored  in  human  memory. 
Broadly  speaking,  a  schema  is  a  collection  of  related  pieces  of  information,  stored  in 
such  a  way  that  when  one  of  the  pieces  is  accessed,  the  others  are  also  readily 
available  for  processing.  Schemas  are  important  because  they  offer  a  way  to 
organize  seemingly  disparate  knowledge.  Thus  facts,  experiences,  general  premises, 
inferences,  procedures,  and  skills  may  all  be  linked  together  as  a  schema.  A  number 
of  psychologists  and  cognitive  scientists  nave  studied  the  schema,  and  there  is 
general  consensus  on  certain  properties  or  characteristics  (see,  for  example.  Van 
Lehn,  1990;  Kintsch  &  Greeno,  1985;  Anderson,  1984;  Rumelhart,  1980;  Baddeley, 
1990;  Skemp,  1987;  and  Marshall,  in  press).  Some  of  these  characteristics  are: 

°  Schema  knowledge  is  tightly  connected,  forming  a  network  of 
relationships. 

°  Schemas  represent  knowledge  of  the  world,  not  just  a  set  of  abstract 
rules. 

° Schemas  are  internally  consistent  and  allow  inferences. 

°A  collection  of  schemas  forms  the  basis  of  an  individual's  knowledge  for 
solving  problems  in  a  given  domain. 

^Schemas  function  as  integrative  tools  or  building  blocks. 

°A  schema  is  the  scaffolding  into  which  new  information  is  assimilated. 

° Schema  organization  of  knowledge  facilitates  memory  search. 


1 


Four  major  types  of  knowledge  play  important  roles  in  a  schema:  feature 
knowledge,  constraint  knowledge,  planning  knowledge,  and  implementation 
knowledge.^  Feature  knowledge  contains  specific  details  about  the  situations  or 
instances  governed  by  the  schema.  Located  here  are  examples,  general  descriptors, 
and  appropriate  analogies.  Constraint  knowledge  serves  to  test  the  current  situation 
to  which  the  schema  is  to  be  applied  against  the  general  features  known  to  be  part  of 
the  schema.  Thus,  it  houses  information  about  whether  the  current  instance  meets 
the  necessary  conditions  of  the  schema.  Planning  knowledge  guides  the  use  of  the 
schema.  Various  actions  may  need  to  be  taken;  subgoals  may  need  to  be  attained. 
Planning  knowledge  identifies  and  orders  these  actions  or  subgoals.  Finally, 
implementation  knowledge  contains  the  rules,  procedures,  and/or  skills  needed  to 
execute  the  actions.  Thus,  this  knowledge  is  needed  to  carry  out  the  plan. 

Instruction  and  Schema  Theory 

Schemas  provide  a  useful  way  of  looking  at  a  subject  area  because  they  can 
serve  as  the  basis  for  organizing  much  of  the  information  pertinent  to  the  area. 
Ideally,  any  domain  would  have  a  relatively  small  number  of  schemas  that  could  be 
used  to  guide  activity  related  to  the  domain.  For  instance,  in  the  domain  of 
arithmetic  word  problems,  five  schemas  appear  to  be  sufficient  (Marshall,  1990;  in 
press). 

A  number  of  different  rationales  may  underlie  instruction.  The  heart  of  the 
matter  is  clarification  of  what  the  instructor  wants  the  learner  to  learn.  On  the  one 
hand,  the  purpose  may  be  for  the  student  to  learn  a  great  many  facts.  Or,  the  goal 
may  be  the  acquisition  of  the  steps  required  of  a  particular  algorithm.  Neither  of 
these  objectives  necessarily  calls  for  schema-based  instruction.  However,  when  the 
aim  is  for  the  student  to  develop  a  body  of  knowledge  organized  in  such  a  way  that 
it  can  be  used  to  solve  real  problems,  then  schema-based  instruction  is  warranted. 

Schema-based  instruction  is  at  least  a  two-tiered  enterprise,  focusing  on 
different  types  of  knowledge  within  one  schema  as  well  as  the  interrelations  among 
several  schemas.  On  the  one  hand,  the  learner  needs  to  acquire  the  components  of 
each  schema  as  outlined  above.  On  the  other,  the  learner  also  needs  to  move  among 
schemas,  using  them  appropriately,  and  when  necessary,  in  combination.  The 
instruction  should  be  designed  explicitly  to  facilitate  both  needs.  Sometimes  this 


^  For  more  elaboration  of  these  knowledge  types,  the  reader  is  referred  to  Marshall, 
forthcoming)  or  to  a  technical  report  available  from  the  author  (Marshall,  Barthuli, 
Brewer,  &  Rose,  1989). 


necessitates  breaking  down  the  subject  matter  into  suitable  chunks.  For  example, 
again  in  the  domain  of  arithmetic  word  problems,  I  have  developed  an  instructional 
system  that  focuses  directly  on  the  knowledge  components  described  above.  In  the 
early  stages  of  instruction,  each  part  of  the  instruction  and  its  accompanying 
exercise  targets  one  of  the  four  knowledge  components.  The  middle  stages 
emphasize  how  these  come  together  for  one  problem  by  focusing  on  the  situation 
expressed  in  the  problem.  The  final  stages  stress  the  relationships  among  different 
situations  (and  hence  different  schemas)  found  in  complex  multi-step  problems.^ 

The  Problem-Solving  Environment  and  Schema  Theory 


One  desires  a  practice  or  testing  environment  in  which  the  learner  can  use 
his  or  her  schema  knowledge  as  it  develops.  In  order  for  each  component  to 
develop  fully,  the  environment  should  supply  a  way  in  which  each  part  of  schema 
knowledge  can  be  used  independently  as  well  as  in  conjunction  with  the  others.  In 
the  word  problem  case,  it  is  not  sufficient  to  present  learners  with  problems  and  ask 
only  for  a  numerical  solution,  because  they  may  not  have  the  schema  components 
sufficiently  developed  to  access  them  independently.  Typical  students  need  at  least 
a  small  amount  of  prompting  to  use  their  knowledge.  Figures  1-4  illustrate  the 
computer  environment  I  and  my  research  assistants  have  developed  to  study  this 
activity.  It  is  a  flexible  system  in  which  the  learner  can  work  methodically  with 
each  part  of  his  or  her  schema  knowledge  for  one  situation  and/or  can  move  globally 
among  different  situations  and  their  associated  schemas. 

An  example  may  clarify  how  this  works.  First,  the  student  receives 
instruction  about  the  different  situations,  learns  to  recognize  a  set  of  icons 
associated  with  them,  and  practices  manipulating  the  icons.  The  icons  are  a  central 
part  of  the  instruction.  Previous  studies  have  indicated  their  importance  in  schema 
development  (Marshall,  1990;  Marshall  &  Brewer,  1990).  When  the  student  has 
developed  the  facility  for  identifying  the  situations  and  for  understanding 
howaspects  of  a  given  situation  map  onto  specific  parts  of  the  appropriate  icon,  he 
or  she  is  introduced  to  the  computer  environment  shown  in  Figures  1-4. 

A  problem-solving  session  starts  with  the  display  of  a  complex  word 
problem  requiring  at  least  two  steps  for  solution.  Figure  1  illustrates  the  computer 
screen  as  it  appears  to  the  student.  Five  icons  are  displayed  immediately  above  the 
problem,  in  the  upper  left  comer.  To  begin  to  solve  the  problem,  the  student  selects 
one  of  the  icons  as  representative  of  a  situation  in  the  problem.  When  selected,  the 


^  Details  of  the  system  are  available  in  Marshall  et  al.,  (1989). 
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Figure  1 :  The  Initial  Screen 


Figure  2:  Screen  After  Icon  Selection 
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Figure  3:  The  Mapping  Option 
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Figure  4:  Illustration  of  Calculator  Option  and  Link 


icon  becomes  highlighted,  and  the  student  can  move  it  into  the  Student  Work  Area 
in  the  lower  left  part  of  the  screen.  At  this  point,  a  pop-up  menu  appears  near  the 
center  of  the  screen  (see  Figure  2).  The  student  will  use  this  menu  to  make  most  of 
his  or  her  choices  during  the  session.  Each  choice  may  be  used  to  characterize  the 
student's  skill  or  ineptitude  and  will  serve  as  a  basis  for  the  assessment  described 
later.  For  the  moment,  I  will  simply  describe  the  choices  open  to  the  student. 

In  solving  the  problem,  the  student  makes  several  decisions.  Among  his 
options  are  to  select  other  icons,  to  examine  the  already-selected  icon  more  closely, 
or  to  ask  for  help.  Thus,  if  the  student  so  wishes,  he  may  select  other  icons  that  he 
considers  pertinent  to  the  problem.  Any  number  may  be  selected.  The  system  does 
not  force  the  student  to  make  particular  choices,  nor  does  it  correct  incorrect 
selections.  A  second  altemauve  is  to  examine  the  chosen  icon  more  closely.  Under 
this  option,  the  selected  icon  is  highlighted  in  the  workspace  and  an  enlarged 
version  of  it  (called  a  diagram  here)  appears  in  the  Mapping  Area  in  the  upper  right 
quadrant  of  the  saeen  (see  Figure  3).  Again,  the  student  has  several  choices.  He 
can  "map"  the  problem  into  the  diagram  (as  partially  shown  in  Figures  3-4).  To  do 
this,  he  goes  to  the  problem  display  on  the  left  side  of  the  screen  and  uses  the  mouse 
to  move  different  phrases,  numbers,  or  expressions  into  the  parts  of  the  enlarged 
diagram.  (In  earlier  instruction,  the  student  has  developed  expertise  in  making  this 
mapping.) 

At  this  point  the  student  may  elect  to  carry  out  a  calculation.  By  selecting 
the  calculation  option  in  the  middle  of  the  right-hand  side  of  the  screen  (as  shown  in 
Figure  3),  the  student  gains  access  to  a  calculator  which  appears  in  the  lower  portion 
(see  Figure  4).  Calculations  may  be  made  with  or  without  mappings.  The  results  of 
calculations  are  movable  and  may  themselves  be  placed  into  the  diagram.  If  the 
learner  believes  the  correct  solution  has  now  been  obtained,  he  may  enter  his 
solution  in  the  Final  Answer  box  at  the  bottom  of  the  screen.  Or,  he  may  return  to 
the  work  area  and  examine  other  icons.  When  the  diagram  is  closed,  all  mapped 
elements  are  remembered  by  the  system,  so  that  if  the  learner  selects  the 
corresponding  icon  again,  the  diagram  automatically  displays  the  pieces  of  the 
problem  and  the  calculations  that  were  previously  entered.  At  all  times,  the  system 
reminds  the  student  which  parts  of  the  icons  have  been  mapped  by  displaying  them 
as  shaded  areas  (see  the  lower  icon  displayed  in  Figure  4). 

Upon  return  to  the  Student  Work  Area,  the  learner  will  now  need  to  select 
another  icon  if  only  one  has  been  previously  chosen.  At  this  point,  he  can  build  an 
explicit  link  between  the  two  icons,  as  shown  in  Figure  4.  This  linkage  indicates 
that  the  contents  of  a  specific  part  of  one  diagram  are  also  the  contents  of  a  specific 
part  of  a  second  diagram.  Values  are  automatically  transferred  and  will  appear  in 
the  diagram  for  the  second  icon. 
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The  learner  thus  moves  back  and  forth  in  the  environment,  looking  at  the 
various  situations  represented  by  the  icons,  and  developing  a  plan  for  solving  the 
problem.  A  fully  developed  plan  would  have  a  set  of  completely  filled  icons  with 
correct  linkages.  Computations  would  have  been  made  at  different  points  of  the 
plan,  with  a  final  computation  yielding  the  answer.  When  the  correct  answer  is 
entered  in  the  Final  Answer  box,  the  system  informs  the  student  that  his  response  is 
correct. 

This  environment  is  an  ideal  arena  for  assessing  schema  knowledge. 
Selection  of  the  appropriate  icon  demonstrates  constraint  knowledge.  The  learner 
recognizes  which  of  the  icons  are  appropriate  for  the  situations  expressed  in  the 
problem.  Mapping  the  parts  of  the  problem  into  the  diagram  uses  feature 
recognition  knowledge.  Each  diagram  corresponds  to  a  particular  situation  and  has 
different  features.  For  example,  there  are  three  characteristics  of  a  change  situation: 
the  quantity  available  before  the  change  occurs,  the  amount  by  which  it  changes, 
and  the  resulting  quantity  after  the  change.  Each  of  these  is  represented  by  a 
separate  part  of  the  change  diagram.  Planning  knowledge  shows  up  in  the  linkages 
among  the  icons  as  well  as  in  the  order  in  which  the  icons  are  accessed.  And, 
finally,  implementation  knowledge  is  evident  through  the  learner's  computations 
and  designation  of  the  final  answer. 

Assessment  and  Schema  Theory 

Schema-based  assessment  has  the  same  focus  as  the  instruction:  the  set  of 
desired  schemas  and  their  individual  knowledge  components.  The  goal  is  to 
examine  the  extent  to  which  learners  have  developed  and  can  use  the  schemas  which 
guided  the  instruction. 

Schema  assessment  presents  some  difficulties  that  do  not  occur  in  other 
types  of  assessment.  A  central  one  is  that  schemas  are  highly  individualized. 
Schemas  are  created  and  tailored  by  individuals  to  reflect  their  own  experiences  and 
understandings  of  the  world.  No  two  individuals  will  form  identical  schemas 
because  no  two  individuals  can  experience  the  world  identically,  but  both  of  them 
may  have  well-formed  and  useful  schemas.  This  contrasts  sharply  with  the 
knowledge  of  a  fact  or  a  computational  procedure  which  all  learners  may  be 
expected  to  Imow  in  the  same  way.  While  it  is  possible  to  assess  knowledge  of  a 
fact  with  a  single  question  that  elicits  the  identical  response  from  all  individuals,  it 
is  not  reasonable  to  apply  the  same  technique  in  assessing  schema  knowledge. 

Nevertheless,  through  schema-based  instruction,  individuals  are  provided 
with  common  experiences  that  facilitate  the  development  of  targeted  schemas. 
Some  of  the  elements  of  any  schema  will  undoubtedly  be  unique  to  each  learner. 
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because  each  one  will  bring  his  or  her  own  prior  experiences,  but  other  pieces  of  the 
schema  should  come  from  the  common  ground  of  instruction. 

The  impact  of  this  individualization  is  that  the  focus  should  not  be  too 
narrow  in  schema  assessment.  Whereas  an  individual  may  not  have  a  specific  piece 
of  information  stored  in  memory  as  part  of  the  schema,  he  may  nonetheless  have  a 
great  deal  of  interrelated  schema  knowledge.  Thus,  satisfactory  assessment  will 
have  a  broader  scope  and  will  look  at  patterns  of  elements  rather  than  single 
elements  of  knowledge.  The  exception  is  the  case  in  which  a  specific  piece  of 
information  is  an  essential  ingredient  of  a  particular  schema.  One  might  want  to 
evaluate  first  that  the  individual  does  indeed  know  this  piece  of  information  and 
second  that  he  associates  it  with  the  correct  schema  and  in  correct  relation  to  other 
requisite  schema  knowledge. 

A  second  unique  characteristic  of  schema  assessment  is  that  it  is  multi¬ 
layered.  One  can  assess  components  within  a  schema,  assess  the  relationships 
among  components  of  a  single  schema,  or  assess  the  interactions  among  a  set  of 
schemas.  All  three  layers  are  considered  in  a  full  assessment. 

On-Line  Assessment 

To  date,  most  assessments  of  schema  knowledge  have  been  through 
interviews  or  thinking-aloud  protocols  (e.g.,  Chi,  Glaser,  Feltovich,  1981;  Marshall, 
1990).  One  reason  that  these  procedures  have  been  widely  utilized  is  that  schema 
assessment  requires  a  great  deal  of  flexibility.  Rarely  can  a  single  question  tap  the 
richness  of  schema  knowledge  possessed  by  an  individual.  Usually  one  begins  with 
a  standardized  question  and  then  allows  a  variety  of  different  follow-up  questions  to 
probe  each  individual's  schema  knowledge. 

The  system  described  in  this  paper  is  one  alternative  to  assessment  by 
interview  for  the  domain  of  arithmetic  word  problems.  It  was  created  in  part  to 
provide  the  means  of  evaluating  students’  schemas  and  in  part  to  give  students  a 
tolerant  environment  in  which  to  practice  their  schema  understanding.  The 
environment  was  described  above.  This  section  outlines  its  capabilities  for 
assessment. 

Perhaps  surprisingly,  a  simple  history  of  the  student's  interaction  with  the 
system  offers  extremely  valuable  information  that  can  be  used  to  assess  schema 
knowledge.  The  system  records  all  students  requests,  responses,  and  mouse  clicks. 
Its  record-keeping  does  not  require  a  great  deal  of  bunslation  and  is  immediately 
readable.  By  recording  the  student's  every  action,  the  system  follows  the  student's 
path  through  the  environment.  The  history  of  that  path  provides  a  great  deal  of 
information  about  how  the  student  perceived  the  problem,  about  any 
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misunderstandings  the  student  had  about  the  problem  and  whether  these  were  self- 
corrected,  and  about  how  the  student  went  about  the  business  of  finding  a  solution. 

Example  of  Student  Responses 

To  illustrate  how  the  word-problem  environment  may  be  used  for 
assessment.  Table  1  contains  the  computer  record  for  one  student  solving  one 
problem.  The  system  output  is  unedited  except  for  the  insertion  of  blank  lines  to 
separate  segments  for  the  accompanying  explanations  in  the  table. 

As  shown  in  Table  1,  the  record  of  student  activity  consists  of  a  list  of 
events.  Each  ev.;nt  corresponds  to  some  action  taken  by  the  student.  The  record  is 
a  history  of  all  events  in  the  order  in  which  they  occurred.  The  primary  events  that 
are  contained  in  a  student  record  are  IconSelected,  Map,  LifiWord,  CalculatorSolve, 
Link,  and  ShowMe.  Brief  descriptions  of  each  one  follow. 

IconSelected.  This  event  indicates  that  the  student  selected  one  of  the  five  icons 
and  placed  it  in  the  Student  Work  Area.  The  name  of  the  corresponding  situation 
and  its  position  in  the  window  are  recorded  as  well  as  its  rank  order  in  the  set  of 
selected  icons.  » 

Map.  Whenever  the  student  requests  that  an  icon  be  enlarged,  this  event  is 
recorded,  together  with  the  name  of  the  diagram  and  its  ranking. 

LiftWord.  This  event  is  recorded  whenever  the  student  selects  a  part  of  the  problem 
or  a  result  of  a  calculation  for  placement  in  a  diagram  or  in  the  final  answer  slot. 
The  record  contains  the  activity,  the  words  or  numbers  that  have  been  lifted,  and  the 
destination  in  which  they  are  placed.  Only  one  map  is  available  at  any  time,  so 
there  is  no  confusion  about  which  diagram  is  designated.  The  final  integer  values  in 
the  event  indicate  the  part  of  the  diagram  used.  Each  component  of  the  diagram  has 
its  own  identification  number. 

CalculatorSolve.  This  event  indicates  that  the  student  requested  a  calculation.  The 
expression  to  be  calculated  and  its  result  are  recorded. 

Link.  The  link  event  has  four  parts:  the  rank  order  of  the  icon  from  which  the  link 
emanates,  the  numbered  part  of  that  icon,  the  rank  order  of  the  icon  in  which  the 
link  terminates,  and  the  numbered  part  of  it. 
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Table  1 

Example  of  System  Assessment  Record' 


SYSTEM  OUTPUT 
(Problem  Ml 6) 


(IconSelected  CHsmall  1  (78 . 285)) 
(IconSelected  GR3small  2  (252 . 248)) 


Map  GR3smali  2) 

(LiftWord  $1200  NIL) 

(LiftWord  $625  4) 

(LiftWord  $350  3) 

(LiftWord  $195  2) 

(CalculatorC  ailed) 

(CalculatorSolve  (625  r350+195)  1170) 
(LiftWord  (Calculation  1170)  1) 


EXPLANATION  OF  OUTPUT 

System  randomly  selects  problem  16 
for  presentation. 

S  selects  the  Change  icon  first  (1)  — 
which  is  the  overall  situation  of  the 
problem  (The  numbers  in 
parentheses  at  the  end  of  the 
statement  indicate  the  location  on  the 
screen  at  which  S  positioned  the 
icon).  S  then  selects  the  Group  icon 
(2)  —  which  is  an  embedded 
subproblem 

S  elects  to  map  the  parts  of  the  Group 
icon.  (At  this  point  the  enlarged 
diagram  would  appear  on  the  upper 
right  portion  of  the  screen.) 

S  first  erroneously  picks  up  the 
number  $1200  and  then  releases  it  — 
presumably  realizing  that  it  was  not 
needed.  S  then  maps  as  subgroups 
the  values  $625,  $350,  and  $195.  S 
calculates  their  sum  and  places  it  in 
the  supergroup  position  of  the 
diagram  S  closes  the  enlarged 
diagram  and  returns  to  the  Student 
Work  Area. 
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Table  1:  continued 


SYSTEM  OUTPUT 


EXPLANATION  OF  OUTPUT 


(Link  2  112) 


(MapCHsmall  1) 

(LiftWord$1200  1) 

(Cal  culatorC  ailed) 
(CalculatorClear(1170-  12)) 
(CalculatorSolve  (12)  12) 
(CalculatorClear  NIL)  •• 
(CalculatorSolve  (1200  -  1 170)  30) 
(LiftWord  (Calculation  30)  3) 
(LiftWord  (Calculation  30)  NIL) 
(LiftWord  (Calculation  30) 

AnswerWindow) 


5  realizes  that  the  supergroup  value 
calculated  for  the  Group  diagram  is 
also  a  value  needed  in  the  Change 
diagram.  S  uses  the  link  command  to 
connect  the  "supergroup  box"  of  the 
Group  icon  and  the  "amount  of 
change  box"  of  the  Change  icon. 

S  then  maps  the  Change  icon,  placing 
the  previously  rejected  value  of  $1200 
into  the  initial  position  of  the 
enlarged  diagram.  The  value  $1170 
already  appears  in  the  map  because 
of  the  explicit  link  made  by  S.  S  does 
several  calculations  at  this  point, 
mostly  incorrect.  S  then  correctly 
calculates  the  value  $30  and  places  it 
first  in  the  diagram  and  then  in  the 
Answer  Box. 


(RightAnswer) 


System  records  that  S  answered  the 
problem  correctly. 


*This  is  an  actual  output  from  one  of  the  students  in  the  study.  The  output  appears  here 
exactly  as  it  is  recorded  by  the  system.  Blank  lines  are  inserted  to  illustrate  the  different 
segments  of  the  problem  solving. 


ShowMe.  Whenever  a  student  asks  for  the  full  display  of  the  solution,  this  event  is 
recorded.  A  student  may  request  the  display  more  than  once.  Every  time  it  is 
requested,  the  event  is  recorded. 

Several  other  events  also  may  be  included  in  the  record.  These  are  found 
less  frequently  and  are  not  described  here. 

Some  Results:  Problem-Solving  Strategies 

Students  using  the  system  have  displayed  a  number  of  different  strategies, 
each  suggesting  a  different  schema  development.  3  The  most  frequent  ones  are 
described  below. 

Before  doing  anything  else,  a  student  may  select  all  icons  that  will  be 
needed.  When  this  is  the  initial  behavior,  we  know  that  the  student  is  using  his 
schema  knowledge  to  construct  a  plan.  A  set  of  subproblems  is  identified,  and  one 
infers  that  the  student  has  mentally  linked  them  together  in  some  way.  There  is 
useful  information  in  the  order  with  which  the  student  selects  these  icons.  The 
student  may  begin  by  selecting  the  first  situation  that  arises  in  the  problem,  moving 
sequentially  through  allmecessary  ones,  and  culminating  with  the  selection  of  the 
overall  situation  that  governs  the  problem.  Such  behavior  represents  a  rather 
groping  approach  and  may  be  contrasted  with  the  selection  first  of  the  overall 
situation  followed  by  the  necessary  subproblem  situations,  an  approach  that 
indicates  awareness  of  the  general  structure  of  the  problem.  These  two  selection 
strategies  show  differences  that  are  roughly  analogous  to  those  of  bottom-up  or  top- 
down  processing.  Generally,  from  the  small  number  of  subjects  we  have  observed, 
we  find  that  students  who  engage  in  the  top-down  approach  (i.e.,  select  the  icon  for 
the  overall  situation  first)  have  a  better  understanding  of  the  problem  and  appear  to 
have  more  coherent  schemas. 

Students  who  were  well  on  their  way  to  having  coherent  schemas  and  using 
them  as  a  related  set  tended  to  begin  their  problem  solving  sessions  in  similar  ways. 
They  made  an  initial  selection  of  one  icon  and  attempted  to  map  it.  If  the  mapping 
was  unsuccessful,  they  closed  the  diagram  and  selected  another  icon,  attempting  to 
map  the  same  problem  elements  as  before.  Thus,  they  demonstrated  self¬ 
monitoring.  Very  rarely  did  any  student  incorrectly  map  part  of  a  problem  and  then 
continue  as  if  the  mapping  were  correct.  If  successful  in  the  initial  mapping  or 
revised  mapping,  the  students  returned  to  the  menu  of  icons  and  selected  a  second 


^  To  date,  the  system  has  been  used  by  a  total  of  31  college  students  with  weak  problem 
solving  skills. 
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icon  with  which  to  work.  At  this  point  we  observed  two  strategies  indicating 
variations  in  students’  understanding.  Some  of  the  students  already  knew  which  part 
of  the  second  icon  contained  an  unknown  quantity  that  corresponded  to  a  part  of  the 
first  icon.  Students  generally  tried  to  link  the  two  icons  together  if  they  were  aware 
of  how  the  components  fit  together.  If  they  were  uncertain,  they  mapped  the  second 
diagram  and  then  carried  out  the  linking.  Only  after  all  mapping  and  linkages  were 
complete  did  students  using  these  strategies  enter  the  answer  in  the  Final  Answer 
window  at  the  bottom  of  the  screen.  They  carefully  went  through  every  step  and 
linkage.  After  solving  several  problems  in  this  way,  they  frequently  switched  to  the 
strategy  described  below  in  which  only  the  first  diagram  was  complete  filled.  This 
switch  suggests  that  students'  use  of  the  schema  knowledge  was  becoming  more 
routinized. 

Other  strategies  are  also  revealing.  Some  students  selected  only  the  icon  for 
the  overall  situation  and  turned  immediately  to  the  mapping  of  the  problem  into  the 
expanded  diagram.  They  mapped  the  appropriate  quantities  ft'om  the  problem  to  the 
diagram  and  also  mapped  the  full  description  of  the  subproblem  (as  stated  in  the 
problem)  directly  into  the  diagram.  They  then  made  the  necessary  calculation  for 
the  subproblem  without  needing  to  map  its  corresponding  diagram.  Students 
displaying  this  strategy , usually  had  a  good  grasp  of  the  overall  and  subproblem 
icons  and  did  not  need  to  have  all  diagrams  displayed.  They  selected  the  correct 
quantities  for  calculation  and  showed  a  high  level  of  confidence  in  what  they  are 
doing.  Moreover,  in  follow-up  interviews  they  indicated  that  they  were  indeed 
using  mental  models  corresponding  to  the  omitted  icons  and  diagrams. 

Another  strategy  was  to  select  a  single  initial  icon,  either  the  same  one  for 
all  problems  or  perhaps  a  random  choice;  in  both  cases,  the  selected  icon  was 
usually  incorrect.  Students  using  this  strategy  went  immediately  to  the  calculator 
without  attempting  to  map  the  problem.  Usually,  they  made  an  incorrect 
calculation,  either  reversing  the  position  of  the  quantities  in  the  calculation  or 
selecting  incorrect  quantities  for  the  calculation.  In  our  sample,  they  also  almost 
always  overlooked  one  of  the  subproblems  required  for  solution.  These  students 
displayed  relatively  weak  schema  knowledge.  Although  they  had  developed  the 
facility  of  identifying  icons  and  mapping  problems  into  the  diagrams  in  previous 
instructional  sessions,  they  elected  not  to  use  their  knowledge  of  the  icons  and/or 
diagrams  and  relied  instead  upon  previously  learned  (and  frequently  incorrect) 
problem-solving  strategies.  * 


^  They  admitted  this  freely  in  followup  interviews.  Most  said  that  they  liked  the 
diagrams  but  already  knew  how  to  solve  the  problems  and  just  used  their  own 
procedures. 
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One  feature  of  the  system  as  currently  implemented  is  a  Show  Me  option. 
At  any  point  in  the  problem  solving,  the  student  can  ask  for  a  display  of  the  full 
solution.  This  appears  in  the  tower  right-hand  portion  of  the  screen  (in  the  same 
location  as  the  calculator).  The  system  begins  by  displaying  the  icon  for  the  overall 
situation  and  then  any  other  needed  ones.  Necessary  links  are  drawn.  The  icon 
corresponding  to  the  most  embedded  subproblem  is  then  enlarged,  displayed  as  a 
diagram,  and  fully  mapped.  However,  no  calculations  are  carried  out.  Show  Me 
simply  displays  the  necessary  icons  and  linkages.  After  it  disappears  from  the 
screen,  the  student  may  elect  to  carry  out  the  solution  or  may  choose  another 
problem.  Most  of  the  students  in  our  sample  elected  to  complete  the  solution. 

Scoring 


A  last  point  about  schema  assessment  is  that  it  is  unlikely  to  yield  a 
numerical  score.  It  does  not  really  matter  how  many  problems  the  student  solved. 
We  are  much  more  concerned  with  the  way  in  which  the  student  went  about  it  and 
the  changes  in  strategies  that  emerge  over  time.  We  take  this  to  be  indicative  of  the 
knowledge  structure  the  student  is  developing.  Alone,  of  course,  the  computer 
assessment  is  not  conclusive.  To  check  its  validity,  we  interviewed  16  of  the  31 
students  following  each  session  to  determine  whether  our  assessment  of  their 
knowledge  based  on  performance  in  the  computer  environment  matches  our 
evaluation  of  schema  development  made  during  the  interview.  Thus  far,  we  find 
strong  agreement  between  the  results  of  the  computer  assessment  and  the  interview 
assessment. 

Conclusions 

There  are  several  important  reasons  for  employing  computer-based 
assessment  as  it  is  presented  here.  First,  it  is  appropriate  for  the  computer-based 
instruction  it  assesses.  Students  are  familiar  with  the  types  of  required  responses 
and  understand  the  idiosyncrasies  of  the  system.  Second,  it  is  unobtrusive. 
Students  may  expect  that  their  responses  are  being  recorded,  but  they  do  not  seem  to 
think  much  about  it.  They  are  free  to  pursue  their  problem  solving  without 
interruptions,  without  being  asked  to  write  down  a  particular  step,  or  without  being 
stopped  to  clarify  what  they  are  now  doing  and  why.  Third,  a  system  such  as  this  is 
economical  because  it  serves  a  dual  purpose.  On  the  one  hand,  it  can  be  an 
instructional  aid,  providing  a  practice  ground  for  individuals  to  hone  their 
understanding  of  the  structure  of  word  problems.  On  the  other,  it  functions  as  an 
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assessment  tool,  allowing  us  to  determine  which  aspects  of  schema  knowledge  are 
well  developed  in  an  individual  and  which  are  weak. 

A  final  advantage  of  this  system  of  assessment  is  that  it  can  provide  both 
diagnostic  information  for  the  individual  and  summative  information  for  a  group  of 
students.  Of  interest  in  the  individual  evaluation  are  questions  such  as  which 
strategy  is  employed  or  whether  the  individual  has  difficulty  with  one  or  more  of  the 
diagrams.  At  the  group  level  of  assessment  we  are  concerned  with  questions  such 
as  determining  whether  the  group  as  a  whole  is  successful  in  solving  complex 
problems  or  whether  most  (or  all)  of  them  demonstrate  a  sufficient  level  of 
competency  in  organizing  their  problem  solving.  Thus,  at  the  group  level,  we  are 
much  less  concerned  with  the  fine-grained  details.  Both  types  of  assessment  are 
typically  needed,  and  this  system  easily  provides  them. 

The  three  issues  of  any  assessment  are  what,  why,  and  how.  What  is 
being  assessed  and  why  it  is  to  be  done  are  the  driving  forces  behind  the 
assessment.  The  issue  of  how  is  not  the  first  question  or  issue  to  be  addressed, 
although  it  is  clearly  an  important  one.  One  of  the  lessons  learned  from  multiple 
choice  tests  is  that  we  should  not  carry  out  a  particular  assessment  just  because  we 
have  a  well -developed  technique.  We  are  very  good  at  developing  multiple  choice 
tests.  We  know  the  psychometric  theory  of  the  testing  situation,  we  know  how  to 
build  certain  estimates;  and  we  can  score  tests  quickly  and  accurately.  However, 
one  of  the  main  problems  facing  assessment  leaders  today  is  that  these  procedures 
are  being  used  in  too  many  instances  because  of  the  third  question-’’how"-and  not 
because  of  the  first  two  questions  just  posed-"whaf’  and  "why”.  It  is  now  widely 
argued  in  assessment  circles  that  a  multiple  choice  test  does  not  reflect  what  an 
individual  understands  and  is  in  fact  a  poor  vehicle  in  many  areas. 

Several  alternatives  to  multiple-choice  tests  are  under  investigation  by  a 
number  of  test  developers.  Among  them  are  things  such  as  portfolios,  performance 
assessments,  and  free-response  items.  Unfortunately,  these  are  viewed  simply  as 
alternatives  to  the  "how"  question,  and  little  attention  has  been  given  to  any  theory 
of  learning  or  understanding  that  should  underlie  them.  As  demonstrated  here,  the 
explicit  linkage  between  instruction,  learning,  and  assessment  is  crucial.  The 
common  underlying  theory  provides  the  basis  for  interpreting  the  assessment. 
Without  it,  one  is  left  only  with  a  collection  of  isolated  student  responses.  Thus, 
schema-based  assessment  puts  the  "what"  and  the  "why"  of  assessment  first,  and  it 
does  so  by  supplying  an  integrated  picture  of  memory,  learning,  and  assessment. 
The  assessment  eventually  made  is  dictated  by  the  model  of  memory  and  learning 
and  not  by  convenience  of  data  collection. 
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